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Missile  Defense  Program  Instability  Affects  Reliability 
of  Earned  Value  Management  Data 


Why  GAO  Did  This  Study 


What  GAO  Found 


By  law,  GAO  is  directed  to  assess 
the  annual  progress  the  Missile 
Defense  Agency  (MDA)  made  in 
developing  and  fielding  the  Ballistic 
Missile  Defense  System  (BMDS). 
GAO  issued  its  latest  assessment  of 
MDA’s  progress  covering  fiscal 
year  2009  in  February  2010.  This 
report  supplements  that 
assessment  to  provide  further 
insight  into  MDA’s  prime 
contractor  performance  for  fiscal 
year  2009.  Prime  contractors  track 
earned  value  management  (EVM) 
by  making  comparisons  that  inform 
the  program  as  to  whether  the 
contractor  is  completing  work  at 
the  cost  budgeted  and  whether  the 
work  scheduled  is  being  completed 
on  time.  Our  analysis  of  contractor 
EVM  data  included  examining 
contract  performance  reports  for 
14  BMDS  contracts,  reviewing  the 
latest  integrated  baseline  reviews, 
performing  extensive  analysis  of 
data  anomalies,  and  conducting 
interviews  with  Defense  Contract 
Management  Agency  (DCMA) 
officials — the  independent 
reviewers  of  MDA  contractor  EVM 
systems. 


What  GAO  Recommends 


GAO  recommends  that  MDA 
resolve  prime  contractor  EVM  data 
reliability  issues  by  the  beginning 
of  fiscal  year  2011.  If,  by  this  time, 
MDA  has  not  resolved  these  issues, 
the  Secretary  of  Defense  should 
provide  a  report  to  Congress  on  the 
steps  MDA  is  taking  to  resolve 
them.  DOD  concurred  with  our 
recommendation. 


Unlike  GAO’s  reports  in  previous  years,  GAO  was  unable  to  analyze  the  EVM 
data  for  all  MDA  contracts.  GAO  determined  that  the  data  for  the  Ground- 
based  Midcourse  Defense  (GMD)  and  Targets  and  Countermeasures  programs 
were  not  sufficiently  reliable  to  include  in  our  report  because  of  instability  in 
these  programs’  baselines.  When  the  baseline  on  which  the  work  is  performed 
and  measured  against  is  no  longer  representative  of  the  program  of  record, 
program  managers  and  other  decision  makers  lose  the  ability  to  develop 
constructive  corrective  action  plans  to  get  the  program  on  track.  Specifically, 
without  reliable  EVM  data,  GAO  was  unable  to  identify  significant 
performance  drivers  or  forecast  future  cost  and  schedule  performance. 
Because  the  two  contracts  associated  with  these  programs  represent  half  of 
the  budgeted  cost  at  completion  for  the  14  contracts  GAO  reviewed,  GAO  also 
determined  it  was  not  appropriate  in  this  report  to  aggregate  total  projected 
underruns  or  overruns  of  the  remaining  12  prime  contracts  as  GAO  has  in 
prior  reports. 

The  GMD  prime  contractor  performance  data  was  not  sufficiently  reliable  to 
use  as  the  basis  for  analysis  because  the  contractor  was  unable  to  update  its 
baseline  to  include  numerous  changes  to  the  program  and  modifications  to 
the  contract.  Despite  three  large  restructures  since  2007  totaling  over  $2 
billion,  the  GMD  program  has  not  conducted  an  integrated  baseline  review 
since  December  2006.  DOD  acquisition  policy  states  that  an  integrated 
baseline  review  is  to  be  conducted  within  6  months  after  contract  award, 
exercise  of  contract  options,  or  a  major  modification  to  an  existing  contract. 
The  Director,  MDA  has  taken  extra  steps  to  gain  insight  into  the  contractor’s 
performance.  Further,  he  intends  to  report  EVM  information  to  Congress 
annually. 

Similarly,  the  EVM  data  for  the  Targets  and  Countermeasures  contractor  is 
also  not  sufficiently  reliable  to  use  in  our  analysis.  DCMA  identified  several 
issues  with  the  stability  of  the  Targets  and  Countermeasures  program  baseline 
including  a  large  amount  of  schedule  and  quantity  changes  to  planned  flight 
tests  and  over  20  contract  changes  to  the  scope  of  work  or  corrective  actions 
to  quality  issues  for  one  of  the  delivery  orders  over  the  course  of  a  year. 
Because  the  contractor  has  not  been  able  to  update  the  established  budget  in 
the  baseline,  the  cost  performance  reports  do  not  reflect  an  appropriate 
baseline  against  which  to  measure  cost  and  schedule  progress. 

Nine  of  the  remaining  twelve  contracts  experienced  cost  overruns  for  fiscal 
year  2009  mostly  because  of  issues  with  maturing  technologies,  immature 
designs,  or  other  technical  issues.  For  example,  the  Airborne  Laser  contractor 
experienced  a  failure  in  some  of  the  system’s  optics  which  required  it  to 
develop  and  procure  new  high  power  optics,  delaying  the  test  schedule  and 
increasing  program  cost. 
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GAO 

^^Accountability  *  Integrity  *  Reliability 

United  States  Government  Accountability  Office 
Washington,  DC  20548 


July  14,  2010 

In  2002,  the  President  of  the  United  States  directed  the  Department  of 
Defense  (DOD)  to  begin  fielding  an  initial  Ballistic  Missile  Defense  System 
(BMDS)  capable  of  defending  the  U.S.  homeland,  deployed  troops,  friends, 
and  allies  against  ballistic  missiles  of  all  ranges  in  all  phases  of  flight.  The 
Missile  Defense  Agency  (MDA),  established  to  develop  and  deploy  these 
missile  defense  capabilities,  began  delivering  an  initial  capability  by 
concurrently  developing  and  fielding  assets.  MDA  is  the  DOD’s  largest 
single  acquisition  program — spending  from  approximately  $7  billion  to 
$9.5  billion  per  year. 

To  more  effectively  manage  complex  investments  such  as  these,  in 
December  1996  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  signed  a  memorandum  announcing  DOD’s  adoption  of 
industry  standards  for  earned  value  management  (EVM)  systems  for  use 
on  defense  acquisitions.1  EVM  is  a  project  management  approach  that,  if 
implemented  appropriately,  provides  objective  reports  of  project  status, 
produces  early  warning  signs  of  impending  schedule  delays  and  cost 
overruns,  and  provides  unbiased  estimates  of  anticipated  costs  at 
completion. 

This  report  provides  an  in-depth  analysis  of  MDA’s  prime  contractor  fiscal 
year  2009  EVM  cost  and  schedule  progress.  Congress  directed  GAO  in  its 
fiscal  year  2002,  2007,  and  2008  National  Defense  Authorization  Acts,  to 
assess  the  cost,  schedule,  testing,  and  performance  progress  that  MDA  is 
making  in  developing  the  BMDS.2  We  have  delivered  assessments  of 
MDA’s  progress  covering  fiscal  years  2003  through  2009,  issuing  our  latest 


1  Earned  value  management  is  a  program  management  tool  that  integrates  the  technical, 
cost,  and  schedule  parameters  of  a  contract.  During  the  planning  phase,  an  integrated 
baseline  is  developed  by  time-phasing  budget  resources  for  defined  work.  As  work  is 
performed  and  measured  against  the  baseline,  the  corresponding  budget  value  is  “earned.” 
Using  this  earned  value  metric,  cost  and  schedule  variances  can  be  determined  and 
analyzed. 

2  National  Defense  Authorization  Act  for  Fiscal  Year  2002,  Pub.  L.  No.  107-107,  §  232(g) 
(2001);  Ronald  W.  Reagan  National  Defense  Authorization  Act  for  Fiscal  Year  2005,  Pub.  L. 
No.  108-375,  §  233  (2004);  National  Defense  Authorization  Act  for  Fiscal  Year  2006,  Pub.  L. 
No.  109-163,  §  232  (2006);  John  Warner  National  Defense  Authorization  Act  for  Fiscal  Year 
2007,  Pub.  L.  No.  109-364,  §  224  (2006);  and  National  Defense  Authorization  Act  for  Fiscal 
Year  2008,  Pub.  L.  No.  110-181,  §  225  (2008). 
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assessment  of  fiscal  year  2009  in  February  2010. 3  As  we  reported  in  our 
February  2010  report,  although  our  annual  assessments  usually  include 
analysis  of  EVM  data  for  MDA  contractors,  this  year  we  are  reporting  on 
this  information  separately. 

To  assess  progress  during  fiscal  year  2009,  we  examined  contract 
performance  reports  for  14  BMDS  contracts  that  were  managed  by  eight 
BMDS  program  elements.4  However,  we  were  only  able  to  report  our 
analysis  of  EVM  data  for  12  of  these  14  contracts  in  appendix  II  due  to 
concerns  with  data  reliability.  The  data  for  the  Ground-based  Midcourse 
Defense  and  the  Targets  and  Countermeasures  contracts  were  not 
sufficiently  reliable  for  inclusion  in  our  analysis.  The  14  contracts  we 
reviewed  EVM  data  for  are: 

•  Aegis  Ballistic  Missile  Defense  (BMD)  weapon  system  software; 

•  two  Aegis  BMD  SM-3  Block  IA  contracts  for 

•  a  fourth  lot  of  27  missiles  and 

•  another  lot  of  24  missiles; 

•  Aegis  BMD  SM-3  Block  IA  and  IB  missile  technology  development  and 
engineering; 

•  Airborne  Laser  (ABL);5 

•  three  Sensors’  contracts 

•  BMDS  radars, 

•  Army  Navy/Transportable  Radar  Surveillance — Model  2  (AN/TPY- 
2)  radar  #7,  and 


3  We  did  not  assess  MDA’s  progress  in  fiscal  year  2002  as  the  agency  did  not  establish  goals 
for  that  fiscal  year.  We  delivered  the  following  reports  for  fiscal  years  2003  through  2007: 
GAO,  Missile  Defense:  Actions  Are  Needed  to  Enhance  Testing  and  Accountability, 
GAO-04-409  (Washington,  D.C.:  Apr.  23,  2004);  Defense  Acquisitions:  Status  of  Ballistic 
Missile  Defense  Program  in  2004,  GAO-05-243  (Washington,  D.C.:  Mar.  31,  2005);  Defense 
Acquisitions:  Missile  Defense  Agency  Fields  Initial  Capability  but  Falls  Short  of 
Original  Goals,  GAO-06-327  (Washington,  D.C.:  Mar.  15,  2006);  Defense  Acquisitions: 
Missile  Defense  Acquisition  Strategy  Generates  Results  but  Delivers  Less  at  a  Higher 
Cost,  GAO-07-387  (Washington,  D.C.:  Mar.  15,  2007);  Defense  Acquisitions:  Progress  Made 
in  Fielding  Missile  Defense,  but  Program,  Is  Short  of  Meeting  Goals,  GAO-08-448 
(Washington,  D.C.:  Mar.  14,  2008);  Defense  Acquisitions:  Production  and  Fielding  of 
Missile  Defense  Components  Continue  with  Less  Testing  and  Validation  Than  Planned, 
GAO-09-338  (Washington,  D.C.:  Mar.  13,  2009);  and  Defense  Acquisitions:  Missile  Defense 
Transition  Provides  Opportunity  to  Strength  Acquisition  Approach,  GAO-10-311 
(Washington,  D.C.:  Feb.  25,  2010). 

4  BMDS  elements  are  separate  ongoing  units  to  address  a  different  facet  of  missile  defense, 
but  that  work  in  unison  to  make  up  the  greater  BMDS. 

5  Since  September  2009,  MDA  has  renamed  this  program  the  Airborne  Laser  Test  Bed. 
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•  Thule  upgraded  early  warning  radar; 

•  Command  and  Control,  Battle  Management,  and  Communications 
(C2BMC); 

•  Ground-based  Midcourse  Defense  (GMD); 

•  Space  Tracking  and  Surveillance  System  (STSS); 

•  Targets  and  Countermeasures;  and 

•  two  Terminal  High  Altitude  Area  Defense  (THAAD)  contracts  for 

•  development  and 

•  fire  unit  fielding  production. 

Because  we  were  only  able  to  analyze  EVM  data  for  12  of  14  contracts,  this 
report  does  not  provide  a  BMDS-level  analysis  as  we  have  provided  in 
previous  years,  again,  because  of  data  reliability  concerns.  To  perform  our 
review  of  the  EVM  data  for  14  contracts,  we  included  several  checks  to 
ensure  data  reliability  including  reviewing  the  most  current 
documentation  on  contractor  performance  data  systems  for  each  of  the 
contractor  sites,  reviewing  the  latest  integrated  baseline  review  (IBR)  for 
each  contract,  and  following  up  with  each  program  office  to  track  how 
identified  risks  are  being  addressed.  In  addition,  we  checked  the  reliability 
of  the  performance  data  by  consulting  with  earned  value  experts  who 
provided  us  with  tools  to  perform  extensive  analysis  to  independently 
review  the  data.  We  followed  up  on  the  results  of  this  analysis  tool  with 
the  program  office  and  further  reviewed  its  responses  with  the  Defense 
Contract  Management  Agency  (DCMA),  which  independently  reviews  all 
of  the  MDA  contractor  EVM  systems.  Our  scope  and  methodology  is 
discussed  in  more  detail  in  appendix  III. 

We  conducted  this  performance  audit  from  February  2010  to  July  2010  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives. 


Background 


MDA’s  mission  is  to  develop  an  integrated  and  layered  BMDS  to  defend  the 
United  States,  its  deployed  forces,  allies,  and  friends.  In  order  to  meet  this 
mission,  MDA  is  developing  a  highly  complex  system  of  systems — land-, 
sea-  and  space-based  sensors,  interceptors  and  battle  management.  Since 
its  initiation  in  2002,  MDA  has  been  given  a  significant  amount  of  flexibility 
in  executing  the  development  and  fielding  of  the  BMDS.  To  enable  MDA  to 
field  and  enhance  a  missile  defense  system  quickly,  the  Secretary  of 
Defense  in  2002  delayed  the  entry  of  the  BMDS  program  into  the  DOD’s 
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traditional  acquisition  process  until  a  mature  capability  was  ready  to  be 
handed  over  to  a  military  service  for  production  and  operation.6  Because 
MDA  does  not  follow  the  traditional  acquisition  process,  it  has  not  yet 
triggered  certain  statutory  and  regulatory  requirements  that  other  major 
defense  acquisition  programs  are  required  to  adhere  to.7 

For  example,  other  major  defense  acquisition  programs  are  required  to 
establish  the  total  scope  of  work  and  total  cost  baselines  as  part  of  their 
entry  into  the  formal  acquisition  cycle.  Title  10  United  States  Code  (U.S.C.) 
section  2435  requires  a  baseline  description  for  major  defense  acquisition 
programs,  however  the  requirement  to  establish  a  baseline  is  not  triggered 
until  a  system  enters  into  system  development  and  demonstration.  DOD 
has  implemented  this  requirement  with  the  acquisition  program  baseline  in 
its  acquisition  policy.8  Because  the  BMDS  has  not  yet  formally  entered  the 
acquisition  cycle,  it  has  not  yet  been  required  to  meet  the  minimum 
requirements  of  section  2435.  Therefore,  because  of  the  Secretary  of 
Defense’s  decision  to  delay  entry  of  the  BMDS  system  into  the  acquisition 
cycle,  MDA  is  not  required  to  establish  the  full  scope  of  work  or  total  cost 
baselines.9  Since  we  began  annual  reporting  on  missile  defense  in  2004,  we 
have  been  unable  to  assess  overall  progress  on  cost.  As  a  result,  one  of  the 
only  tools  available  for  us  to  use  in  assessing  BMDS  costs  is  the  costs 
reported  on  individual  contracts. 


6  Secretary  of  Defense  Memorandum  “Missile  Defense  Program  Direction”  (Jan.  2,  2002). 

'  The  BMDS  program  meets  the  definition  of  a  major  defense  acquisition  program,  which  is 
defined  in  10  U.S.C.  §  2430. 

8  DOD  Directive  5000.01  (May,  2003)  and  DOD  Instruction  5000.02  (Dec.  2008). 

9  Though  MDA  is  not  yet  required  to  meet  the  requirements  to  establish  a  baseline  under  10 
U.S.C.  §  2435,  Congress  has  enacted  legislation  requiring  MDA  to  establish  some  baselines. 
The  Fiscal  Year  2005  National  Defense  Authorization  Act,  Pub.  L.  No.  108-375,  §  234(e), 
required  the  Director,  MDA,  to  establish  and  report  annually  to  Congress  a  cost,  schedule, 
and  performance  baseline  for  each  block  configuration  being  fielded.  MDA  has  since 
terminated  its  block  approach.  In  addition,  the  National  Defense  Authorization  Act  for 
Fiscal  Year  2008,  Pub.  L.  No.  110-181,  §  223(g)  required  that  no  later  than  the  submittal  of 
the  budget  for  fiscal  year  2009,  MDA  shall  “establish  acquisition  cost,  schedule  and 
performance  baselines”  for  BMDS  elements  that  have  entered  the  equivalent  of  system 
development  and  demonstration  or  are  being  produced  and  acquired  for  operational 
fielding. 
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MDA  employs  prime  contractors  to  accomplish  different  tasks  that  are 
needed  to  develop  and  field  the  BMDS.  Prime  contractors  receive  the  bulk 
of  funds  MDA  requests  each  year  and  work  to  provide  the  hardware  and 
software  for  elements  of  the  BMDS. 

Table  1  provides  a  brief  description  of  eight  BMDS  elements  and  the  prime 
contracts  associated  with  these  elements  currently  under  development  by 
MDA. 


Table  1 :  MDA’s  BMDS  Elements  and  Prime  Contracts 

BMDS  element 

Element  description 

Prime  contracts  associated 
with  element 

Contractor 

Aegis  BM  D 

Aegis  BMD  is  a  ship-based 
missile  defense  system  designed 
to  destroy  short-  to  intermediate- 
range  ballistic  missiles  during  the 
midcourse  phase  of  its  flight. 

Aegis  Weapon  System  contract 
for  software  systems 

Lockheed  Martin  Mission  Systems 
and  Sensors 

Two  Aegis  BMD  contracts  for  the 
production  of  SM-3  Block  IA 
missiles  for  a  fourth  lot  of  27 
missiles  and  another  lot  of  24 
missiles 

Raytheon 

Aegis  BMD  SM-3  Block  IA  and  IB 
missile  technology  development 
and  engineering  contract  for 
development  efforts  on  the  Aegis 
BMD  SM-3  Block  IB  missile,  the 
production  of  one  SM-3  Block  IB 
flight  test  missile,  and  efforts  to 
supportSM-3  Block  IA 
sustainment  engineering  and  flight 
test  support 

ABL 

ABL  is  an  aircraft-based  missile 
defense  system  designed  to 
destroy  all  classes  of  ballistic 
missiles  during  the  boost  phase  of 
their  flight.  ABL  employs  a  high- 
energy  chemical  laser  to  rupture  a 
missile's  motor  casing,  causing 
the  missile  to  lose  thrust  or  flight 
control. 

ABL  contract  to  build  and  test  the 
airborne  laser  weapon  system 

Boeing 
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BMDS  element 

Element  description 

Prime  contracts  associated 
with  element 

Contractor 

C2BMC 

C2BMC  is  the  integrating  element 
of  the  BMDS.  Its  role  is  to  provide 
deliberate  planning,  situational 
awareness,  sensor  management, 
and  battle  management  for  the 
integrated  BM  DS .  C2BMC 
delivers  hardware  and  software 
capabilities  in  spiral  development 
drops— the  current  operational 
delivery  is  Spiral  6.2  and  the 
program  is  currently  developing 

S  piral  6.4  planned  for  delivery  in 
the  first  quarter  of  fiscal  year 

2011. 

C2BMC  contract  to  develop,  test, 
and  field  an  integrating  system  for 
the  BMDS 

Lockheed  Martin  Information 
Systems  &  Global  Services 

GMD 

GMD  is  a  ground-based  missile 
defense  system  designed  to 
destroy  intercontinental  ballistic 
missiles  during  the  midcourse 
phase  of  their  flight.  Its  mission  is 
to  protect  the  U.S.  homeland 
against  ballistic  missile  attacks 
from  North  Korea  and  the  Middle 
East.  MDA  is  planning  on 
emplacing  30  operational 
interceptors  at  F ort  G  reely, 

Alaska,  and  Vandenberg  Air 

Force  Base,  California,  by  the  end 
of  fiscal  year  2010. 

GMD  contract  to  develop  and 
deploy  the  ground-based  system 

Boeing 

Sensors 

MDA  is  developing  various  stand¬ 
alone  radars  for  fielding.  These 
include  forward-based  sensors; 
mobile,  sea-based  sensors,  such 
as  the  Sea-based  X-band  Radar; 
and  upgrades  to  existing  early- 
warning  radars.  The  BMDS  uses 
these  sensors  to  identify  and 
continuously  track  ballistic 
missiles  in  all  phases  offlight. 

BMDS  radars  contract  for  the 
development  and  production  for 
four AN/TPY-2  radars 

Contract  for  the  AN/TPY-2  radar 
#7,  a  forward-based  radar  to  be  a 
part  of  the  second  TH  AAD  battery 

Contract  for  the  Thule  radar,  an 
upgraded  early  warning  radar 

Raytheon 

STSS 

The  STSS  is  designed  to  acquire 
and  track  threat  ballistic  missiles 
in  all  stages  of  flight  through  the 
development  and  launch  of  two 
low-orbit  demonstration  satellites. 

In  fiscal  year  2009,  MDA 
successfully  launched  both 
satellites.  Over  the  next  2  years, 
the  two  satellites  will  take  partin  a 
series  of  tests  to  demonstrate 
their  functionality  and 
interoperability  with  the  BMDS. 
There  is  no  operational  system 
planned  for  STSS . 

STSS  contract  for  development  of 
demonstration  satellites 

Northrop  Grumman  Aerospace 
Systems 
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BMDS  element 

Element  description 

Prime  contracts  associated 
with  element 

Contractor 

Targets  and 
Countermeasures 

MDA  develops  and  provides  a 
series  of  targets  used  in  BMDS 
flight  tests  to  present  authentic 
threat  scenarios.  The  targets  are 
designed  to  encompass  the  full 
spectrum  of  threat  missile  ranges 
and  capabilities. 

Targets  and  Countermeasures 
contract  structured  by  delivery 
orders  to  develop  targets 
including  short-range  targets, 
medium-range  targets,  and 
intermediate-range  targets 
including  the  Flexible  Target 
Family's  launch  vehicle-2.  In 
addition,  the  program  develops 
air-launched  targets  for  various 
ranges 

Lockheed  Martin  Space  Systems 

THAAD 

THAAD  is  a  ground-based  missile 
defense  system  designed  to 
destroy  short-  and  medium-range 
ballistic  missiles  during  the  late- 
midcourse  and  terminal  phases  of 
flight.  A  THAAD  Battery  is  made 
up  of  3  to  6  launchers,  24  to  48 
interceptors,  1  fire  control,  1 
radar,  and  peculiar  support 
equipment. 

THAAD  development  contract  to 
develop  the  hardware  and 
software  to  conduct  ground  and 
flight  testing  to  validate  and  verify 
the  design  of  the  THAAD  weapon 
system.  In  addition,  this  contract 
produced  a  fire  control  and 
launcher  for  the  firstTHAAD 
battery 

THAAD  fire  unit  fielding 
production  contract  to  produce 
components  of  THAAD  Battery  #1 
and  #2— except  for  the  one 
launcher  and  fire  control  provided 
by  the  development  contract  and 
the  radars  that  are  produced  by 
the  Sensors  contractor 

Lockheed  Martin  Space  Systems 
Company 

Source:  MDA  data. 


Each  BMDS  program  office’s  prime  contractor  provides  monthly  earned 
value  reports  which  provide  insight  into  the  dollar  gained  or  lost  for  each 
dollar  invested.  These  Contract  Performance  Reports  compare  monthly 
progress  to  the  existing  cost  or  schedule  performance  baseline  to  reveal 
whether  the  work  scheduled  is  being  completed  on  time  and  if  the  work  is 
being  completed  at  the  cost  budgeted.  For  example,  if  the  contractor  was 
able  to  complete  more  work  than  scheduled  and  for  less  cost  than 
budgeted,  the  contractor  reports  a  positive  schedule  and  cost  variance,  or 
“underrun”.  Alternatively,  if  the  contractor  was  not  able  to  complete  the 
work  in  the  scheduled  time  period  and  spent  more  than  budgeted,  the 
contractor  reports  both  a  negative  schedule  and  cost  variance,  or 
“overrun”.  The  results  can  also  be  mixed  by,  for  example,  completing  the 
work  ahead  of  schedule  (a  positive  schedule  variance)  but  spending  more 
than  budgeted  to  do  so  (a  negative  cost  variance). 

We  also  used  contract  performance  report  data  to  base  projections  of 
likely  overrun  or  underrun  of  each  prime  contractor’s  budgeted  cost  at 
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completion.  Our  projections  of  overruns  or  underruns  to  the  budgeted  cost 
at  completion  are  based  on  the  assumption  that  the  contractor  will 
continue  to  perform  in  the  future  as  it  has  in  the  past.  In  addition,  since  the 
budgeted  cost  at  completion  provides  the  basis  for  our  projected  overruns, 
we  also  provide  it  for  each  contract  we  assessed  in  appendix  II.10 

In  addition,  as  part  of  the  yearly  system  compliance  verification  process, 
DCMA  conducts  a  periodic  surveillance  of  contractor  EVM  systems  to 
determine  initial  and  continuing  compliance  of  those  management  systems 
with  government  accepted  standards. 11  Surveillance  (routine  evaluation 
and  assessment)  of  the  EVM  systems  is  mandatory  for  all  contracts  that 
require  EVM  systems  compliance. 12  Surveillance  ensures  that  the 
contractor  is  meeting  contractual  terms  and  conditions  and  is  in 
compliance  with  applicable  policies  and  regulations.  DCMA  has  primary 
responsibility  for  surveillance  of  the  prime  contractor  and  sub-tier 
suppliers  with  EVM  requirements.  According  to  a  DCMA  Earned  Value 
Management  Center  official  responsible  for  leading  system  surveillance,  at 
the  completion  of  the  assessment,  the  DCMA  Earned  Value  Management 
Center  submits  to  the  contracting  officer  a  status  of  the  contractor’s  EVM 
system  compliance,  including  all  supporting  data  to  that  effect. 

If  deficiencies  are  found  during  the  course  of  the  surveillance  process,  it  is 
the  surveillance  team’s  responsibility,  working  through  DCMA’s  Earned 
Value  Management  Center,  to  issue  a  written  corrective  action  request. 

The  purpose  of  a  corrective  action  request  is  to  formally  notify  the 
contractor  that  a  documented  course  of  action  in  the  form  of  a  corrective 


10  The  budget  at  completion  represents  the  total  planned  value  of  the  contract. 

11  American  National  Standards  Institute/Electronics  Industries  Alliance  -748  is  a  collection 
of  32  earned  value  management  system  guidelines  that  incorporate  business  best  practices 
for  program  management  systems  proven  to  provide  strong  benefits  for  program  or 
enterprise  planning  and  control.  The  processes  include  integration  of  program  scope, 
schedule,  and  cost  objectives,  establishment  of  a  baseline  plan  for  accomplishment  of 
program  objectives,  and  use  of  earned  value  techniques  for  performance  measurement 
during  the  execution  of  a  program.  The  system  provides  a  sound  basis  for  problem 
identification,  corrective  actions,  and  management  replanning  as  required. 

12  DOD’s  Earned  Value  Implementation  Guide  states  that  surveillance  of  management 
control  systems  is  required  for  all  contract  efforts  that  require  EVM  compliance  with 
American  National  Standards  Institute/Electronics  Industries  Alliance  -748.  According  to 
the  Defense  Federal  Acquisition  Regulation  Supplement,  contractors  with  cost  and 
incentive  contracts  with  values  over  certain  thresholds  shall  use  an  earned  value 
management  system  that  complies  with  the  32  earned  value  management  system  guidelines 
established  by  American  National  Standards  Institute/Electronics  Industries  Alliance  -748. 
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action  plan  is  needed  to  bring  the  EVM  system  in  compliance  with 
government  accepted  EVM  system  guidelines.  Corrective  action  requests 
range  in  severity  from  Level  I  to  Level  IV  where,  according  to  a  DCMA 
Earned  Value  Management  Center  official  responsible  for  leading  system 
surveillance,  Level  I  is  for  non-compliance  with  the  Defense  Federal 
Regulation  Acquisition  Supplement  clauses  in  the  contract  that  can  be 
corrected  immediately  and  for  which  no  special  management  attention  is 
required,  and  Level  IV  identifies  issues  where  cost,  schedule,  technical 
performance,  resources,  or  management  process  issues  have  unfavorably 
affected  the  supplier's  EVM  so  that  it  is  incapable  of  reporting  meaningful 
EVM  across  multiple  programs  or  multiple  sites;  and  these  issues  have  not 
been  corrected.  Level  III  and  IV  corrective  action  requests  may  trigger 
formal  reviews  such  as  post  award  review  for  cause,  compliance  reviews, 
or  other  system  validation  reviews  and  may  result  in  suspension  or 
revocation  of  EVM  systems  certification. 


EVM  Data  for  the 
GMD  and  Targets  and 
Countermeasures 
Programs  Are  Not 
Sufficiently  Reliable 


For  GMD  and  Targets  and  Countermeasures,  we  determined  that  the  EVM 
data  were  not  sufficiently  reliable  to  analyze  these  contracts’  cost  and 
schedule  performance  because  of  instability  in  these  programs.  Without 
reliable  EVM  data,  we  are  unable  to  identify  significant  performance 
drivers  or  forecast  future  cost  and  schedule  performance.  Further,  when 
the  baseline  on  which  the  work  is  performed  and  measured  against  is  no 
longer  representative  of  the  program  of  record,  program  managers  and 
other  decision  makers  lose  the  ability  to  develop  constructive  corrective 
action  plans  to  get  the  program  on  track.  These  reliability  issues  affect 
MDA’s  oversight  of  contractor  progress  and  both  MDA  and  GAO’s  ability 
to  report  this  progress  to  external  parties  and  Congress.  MDA  officials 
were  aware  that  significant  changes  were  not  reflected  in  the  baselines  for 
these  two  elements  and  have  been  conducting  more  extensive  oversight  to 
compensate,  but  did  not  alert  us  to  this  issue  during  the  course  of  our 
audit.  The  Director,  MDA  has  acknowledged  the  importance  of  EVM  and 
to  address  some  of  these  issues  he  has  enacted  quarterly  reviews  of  each 
of  the  program’s  baselines.  Further,  he  intends  to  report  EVM  information 
to  Congress  annually. 


According  to  DCMA  officials,  there  were  several  issues  associated  with  the 
Boeing  EVM  system  for  GMD.  One  of  the  main  issues  was  the  contractor’s 
inability  to  maintain  a  consistent  performance  measurement  baseline. 

With  numerous  changes  to  the  program  and  modifications  to  the  contract, 
the  contractor  experienced  difficulty  incorporating  these  changes  into  the 
baseline  in  order  to  measure  performance  against  this  new  work.  For 
example,  although  the  GMD  program  experienced  a  $1.3  billion  dollar 
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restructure  in  2007,  another  major  restructure  beginning  in  fiscal  year  2008 
for  over  $500  million  that  was  completed  in  fiscal  year  2009, 13  and  a  third 
in  fiscal  year  2010  for  over  $380  million,  the  GMD  program  has  not 
conducted  an  IBR 14  since  December  2006.15DOD’s  acquisition  policy  states 
that  an  IBR  is  to  be  conducted  within  6  months  after  contract  award, 
exercise  of  contract  options,  or  major  modifications  to  a  contract. 16  DCMA 
officials  told  us  that  the  GMD  program  had  an  IBR  underway  following  the 
restructure  that  began  in  fiscal  year  2008  and  completed  in  fiscal  year 
2009,  but  in  May  2009  the  program  was  again  redirected  and  the  baseline 
review  was  cancelled. 17 

The  Director,  MDA  explained  that  some  of  the  GMD  program’s  baseline 
instability  from  frequent  restructures  was  related  to  the  changing  GMD 
role  in  European  defense.  The  February  2007  budget  request  for  fiscal  year 
2008  included  an  approach  to  European  defense  focused  on  GBIs  from  the 
GMD  element  and  a  large  fixed  radar  as  well  as  transportable  X-Band 
radars.  In  September  2009,  the  administration  altered  its  approach  to 
European  defense  and  instead  constructed  a  defense  system  to  consist 
primarily  of  Aegis  BMD  sea-based  and  land-based  systems  and 
interceptors,  as  well  as  various  sensors  to  be  deployed  over  time  as  the 
various  capabilities  are  matured.  The  Director  told  us  that  these  European 


13  MDA  notes  that  the  two  major  program  restructures  in  2007  and  fiscal  year  2008  were 
accomplished  via  an  alpha  contracting  process.  During  this  process,  there  is  joint 
government  and  contractor  participation  including  agreement  on  scope  and  requirements 
development,  integrated  schedules,  and  amount  and  time  phasing  of  resources.  Although 
these  activities  satisfy  some  IBR  objectives,  alpha  contracting  is  not  a  substitute  for 
conducting  an  IBR. 

14  An  integrated  baseline  review  verifies  the  technical  content  of  the  baseline.  It  also 
ensures  that  contractor  personnel  understand  and  have  been  adequately  trained  to  collect 
EVM  data.  In  addition,  the  review  verifies  the  accuracy  of  the  related  budget  and  schedules, 
ensures  that  risks  have  been  properly  identified,  assesses  the  contractor’s  ability  to 
implement  proper  earned  value  management,  and  determines  if  the  work  identified  by  the 
contractor  meets  the  program’s  objectives.  The  government’s  program  manager  and 
technical  staff  carry  out  this  review  with  their  contractor  counterparts. 

15  The  GMD  program  has  conducted  integrated  baseline  reviews  on  a  subcontract  effort  at 
the  Fort  Greely  Power  Plant,  but  has  not  conducted  a  comprehensive  integrated  baseline 
review  of  the  contract. 

ltJ  Department  of  Defense  Instruction,  5000.02  (Dec.  2,  2008). 

17  The  MDA  Director  told  us  that  the  IBR  was  canceled  because  the  proposed  budget 
changes  would  have  reduced  the  program’s  budget  by  nearly  half.  Later,  the  program’s 
funding  was  restored  and  a  subsequent  restructure  was  issued  in  October  2009. 
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capability  requirements  changes  drastically  affected  the  GMD  program  as 
a  significant  amount  of  work  had  to  be  restructured. 

During  these  three  to  four  years  of  GMD  baseline  instability,  the  Director, 
MDA  told  us  that  MDA  took  steps  to  gain  additional  insight  into  the 
contractor’s  progress.  The  program  held  added  reviews  in  the  absence  of 
IBRs  to  understand  planned  near-term  effort  and  how  well  they  were 
executing  against  those  plans.  In  addition,  the  Director  told  us  that  the 
program  held  monthly  focus  sessions  during  which  the  joint  government 
and  contractor  teams  briefed  the  status  of  progress  and  risks.  The  Director 
acknowledged  that  these  insights  are  necessary  to  understand  the  meaning 
of  the  near-term  EVM  data.  However,  without  the  benefit  of  a  documented 
IBR  after  multiple  larger  restructures  to  the  program  or  being  made  aware 
of  MDA’s  added  reviews,  we  do  not  have  sufficient  confidence  in  the  GMD 
program  performance  measurement  baseline  to  reliably  analyze  the 
existing  EVM  data. 

Boeing  and  MDA  are  taking  steps  to  address  problems  with  the  reliability 
of  the  contractor’s  EVM  data.  The  contractor  had  planned  to  deliver  a 
performance  measurement  baseline  by  May  2010  and  the  GMD  program  is 
planning  to  conduct  a  series  of  IBRs  on  the  remaining  prime  and  major 
subcontractor  effort  beginning  in  July  2010.  In  addition,  the  contractor  is 
taking  initiatives  to  put  a  performance  measurement  baseline  in  place  as 
quickly  as  possible  and  is  providing  additional  training  for  its  management 
and  control  account  managers  in  charge  of  EVM.  The  Director,  MDA  told 
us  that  MDA  was  changing  how  its  future  contracts  for  the  GMD  program 
are  being  structured  to  be  more  receptive  to  modifications.  This  new 
contract  structure  will  include  dividing  the  work  into  delivery  orders  so 
that  modifications  will  be  reflected  at  a  delivery  order  level  instead  of 
affecting  a  larger  contract.  These  steps  may  help  resolve  the  EVM  issues; 
however  we  cannot  determine  the  full  effect  of  these  steps  until  further 
evaluation  after  their  full  implementation. 

Similarly,  we  have  determined  that  the  EVM  data  for  the  Targets  and 
Countermeasures  contractor,  Lockheed  Martin  Space  Systems,  are  not 
sufficiently  reliable  for  inclusion  in  our  analysis.  Based  on  discussions 
with  and  reports  issued  by  DCMA,  the  Targets  and  Countermeasures 
contractor  was  unable  to  update  its  baseline  because  of  numerous 
program  changes.  In  September  2007,  when  the  delivery  order  for  the 
launch  vehicle-2  was  approximately  60  percent  complete,  Lockheed  Martin 
signaled  that  its  baseline  was  no  longer  valid  by  requesting  a  formal 
reprogramming  of  the  effort  to  include  an  overrun  in  its  baseline  for  this 
delivery  order.  MDA  allowed  the  contractor  to  perform  a  schedule 
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rebaseline  and  remove  schedule  variances  -  but  did  not  provide  any  more 
budget  for  the  recognized  overrun  in  the  performance  measurement 
baseline.  As  a  result,  DCMA  reported  that  the  performance  indicators  for 
this  delivery  order,  needed  to  estimate  a  contract  cost  at  completion,  were 
unrealistic.  According  to  the  Director,  MDA  did  not  believe  the  contractor 
had  justified  that  there  was  a  scope  change  warranting  additional  budget 
in  the  performance  measurement  baseline.  He  said  he  believed  doing  so 
would  mask  problems  the  contractor  was  experiencing  planning  and 
executing  the  contract  which  he  identified  as  the  issue  as  opposed  to 
changes  in  the  contract’s  scope.  According  to  the  Director,  one  example  of 
the  issues  the  contractor  was  experiencing  on  this  delivery  order  included 
a  failure  rate  of  64  percent  on  production  qualification  components.  MDA 
has  since  completed  the  work  on  this  delivery  order  and  begun  managing 
follow-on  target  production  on  a  newly  established  delivery  order. 

In  addition,  during  fiscal  year  2009  DCMA  identified  several  issues  with  the 
stability  of  the  Targets  and  Countermeasures  program  baseline.  For 
example,  program  changes  since  fiscal  year  2008  on  one  delivery  order 
included  over  20  contract  changes  to  the  scope  of  work  or  corrective 
actions  to  quality  issues.  In  addition,  the  schedule  and  quantity  of  planned 
flight  tests  changed  significantly.  During  the  fiscal  year,  DCMA  submitted  a 
corrective  action  request  for  noncompliance  with  incorporating  authorized 
changes  in  a  timely  manner  although  the  contractor  was  able  to  close  this 
issue  before  the  end  of  the  reporting  period.  Because  of  the  instability  in 
the  baseline  and  the  contractor’s  inability  to  update  the  baseline  with 
these  frequent  changes,  we  determined  the  cost  performance  reports  for 
2009  do  not  reflect  an  appropriate  baseline  against  which  to  measure  cost 
and  schedule  progress. 

According  to  the  Director,  MDA,  the  agency  has  undertaken  a  major  effort 
to  stabilize  the  Targets  and  Countermeasures  program.  MDA  has 
established  a  new  target  acquisition  strategy  to  address  recurring  target 
performance  issues  and  increases  in  target  costs.  In  this  new  strategy,  the 
agency  will  buy  generic  targets  in  larger  lots  that  are  not  tied  to  a 
particular  test  instead  of  smaller  lots.  This  effort  should  also  help  increase 
MDA’s  flexibility  to  respond  to  changing  program  requirements.  In 
addition,  the  Director,  MDA  told  us  that  the  Director  of  Engineering  at 
MDA  will  define  target  requirements  instead  of  the  program  manager 
which  should  also  help  create  more  stability. 


Page  12 


GAO-10-676  Missile  Defense  Contractor  Performance 


Despite  Non-Compliance 
Ratings  for  MDA  Prime 
Contractor  EVM  Systems, 
Most  Were  Sufficiently 
Reliable  for  GAO  Review 


During  the  course  of  our  review,  we  found  that  DCMA  assessed  7  of  the  14 
contractors’  EVM  systems  as  noncompliant  in  fiscal  year  2009.  DCMA  also 
rated  3  of  the  14  contractors  systems  as  unassessed.  We  reviewed  the 
basis  for  the  noncompliance  and  unassessed  ratings  and  determined  that 
only  the  GMD  and  Targets  and  Countermeasures  contractor  EVM  issues 
affected  the  reliability  of  the  data  for  our  purposes.  See  table  2  for  the 
DCMA  compliance  ratings  for  the  14  MDA  prime  contracts’  EVM  systems 
and  GAO’s  reliability  assessment. 


Table  2:  DCMA  Compliance  Rating  and  GAO  Reliability  Assessment  for  MDA  Prime  Contractor  EVM  Systems 

Contractor  site  EVM  system 
compliance  rating  for  2009 

GAO  determination 

Contracts 

Not  assessed  Compliant  Noncompliant 

Not 

sufficiently  reliable 

Aegis  BMD  Weapon  System 

X 

Aegis  BMD  SM-3  Block  IA  missiles  forfourth  lot  of  27 
missiles 

X 

Aegis  BMD  SM-3  Block  IA  missiles  for  another  lot  of 

24  missiles 

X 

Aegis  BMD  SM-3  Block  IA  and  IB  missile  technology 
developmentand  engineering 

X 

ABL 

X 

C2BMC 

X 

GMD 

X 

X 

Sensors'  BMDS  Radars 

X 

Sensors'  AN/TPY-2  radar  #7 

X 

Sensors'  Thule  radar 

X 

STSS  contract 

X 

Targets  and  Countermeasures 

X 

X 

THAAD  development 

X 

THAAD  fire  unit  fielding  production 

X 

Source:  DCMA  (data);  GAO  (presentation). 

Note:  A  rating  of  noncompliant  indicates  that  at  least  one  corrective  action  requestwas  open  atthe 
end  of  the  rating  assessmentperiod.  The  noted  noncompliance  can  vary  significantly  from  a  small 
isolated  case  that  does  not  affect  management  data  reported  to  being  systemic  across  the  company 
affecting  all  management  data  reported. 

Five  EVM  systems  besides  the  GMD  and  Targets  and  Countermeasures 
contractor  EVM  systems  were  rated  as  noncompliant  by  DCMA  during  the 
fiscal  year  but  did  not  lead  to  GAO  to  conclude  that  the  EVM  data  were  not 
sufficiently  reliable.  In  order  to  judge  the  reliability  of  the  data,  we 
reviewed  the  significance  of  any  open  corrective  action  request(s)  that 
triggered  a  noncompliance  rating  and  its  impact  on  the  contractor’s  ability 
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to  judge  cost  and  schedule  performance  against  a  baseline.  During  the 
course  of  our  audit,  we  interviewed  DCMA  representatives  at  each  of  the 
contractor  sites  to  understand  the  basis  for  the  noncompliance 
determination  and  to  gain  information  to  help  us  assess  the  reliability  of 
the  data. 

For  example,  the  EVM  system  of  the  STSS  contractor  Northrop  Grumman 
was  deemed  noncompliant  because  of  two  low-level  corrective  action 
requests  related  to  issues  with  other  contracts  that  did  not  materially 
affect  the  performance  baseline  for  the  STSS  contract  we  assessed.  Also, 
the  C2BMC’s  contractor  Lockheed  Martin  Information  Systems  &  Global 
Services  received  a  rating  of  noncompliant  during  2009  because  of  a 
corrective  action  request  that  stated  that  major  subcontractor  efforts  were 
not  specifically  identified,  assigned,  or  tracked  in  the  organizational 
breakdown  structure.  However,  after  the  noncompliant  rating  was  given, 
DCMA  reversed  its  decision  and  decided  to  close  the  corrective  action 
without  requiring  the  contractor  to  change  its  methods. 

In  addition,  although  DCMA  was  unable  to  assess  two  EVM  systems  during 
2009  for  Lockheed  Martin  Mission  Systems  and  Sensors  under  the  Aegis 
BMD  weapon  system  contract,  and  Lockheed  Martin  Space  Systems 
Company  under  the  two  THAAD  contracts,  we  determined  that  the 
reasons  for  the  unassessed  rating  did  not  lead  to  issues  with  data 
reliability.  According  to  the  DCMA  EVM  specialist  responsible  for 
monitoring  the  Aegis  BMD  weapon  system,  the  Aegis  BMD  weapon  system 
contractor  was  unassessed  because  some  of  the  accounting  guidelines 
could  not  be  assessed  in  time  for  the  compliance  rating.  In  addition,  the 
THAAD  contractor  was  not  assessed  because,  according  to  DCMA, 
although  the  contractor  had  addressed  the  open  corrective  action 
requests,  DCMA  did  not  have  the  resources  to  review  and  document  the 
effectiveness  of  those  actions  in  order  to  close  these  items  before  the  end 
of  the  rating  assessment  period.  However,  subsequent  to  the  closing  of  the 
rating  assessment  period,  the  contractor’s  actions  were  deemed  sufficient 
by  DCMA  to  fix  the  unresolved  issues  and  the  corrective  action  requests 
were  closed. 
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BMDS  Prime 
Contractors 
Aggregate  Analysis 
Not  Appropriate  Due 
to  Data  Reliability 
Issues 


We  are  unable  in  this  year’s  report  to  aggregate  total  projected  underruns 
or  overruns  in  our  analysis  of  the  remaining  12  prime  contracts  because 
we  had  to  exclude  the  GMD  and  Targets  and  Countermeasures  programs 
due  to  data  reliability  issues.  The  GMD  and  Targets  and  Countermeasures 
prime  contracts’  budgeted  costs  at  completion  total  nearly  $16  billion 
dollars  or  half  of  the  total  14  contracts’  budgeted  cost  at  completion.  By 
removing  such  a  large  portion  of  data  from  our  analysis,  we  determined 
that  it  is  inappropriate  to  perform  any  aggregate  analysis.  More  detail  is 
provided  for  each  of  the  contractors  responsible  for  the  remaining  twelve 
BMDS  contracts’  cost  and  schedule  performance  in  appendix  II. 


Nine  of  the  remaining  12  contracts  experienced  cost  overruns  for  fiscal 
year  2009.  Most  of  the  overruns  were  because  of  issues  with  maturing 
technologies,  immature  designs,  or  other  technical  issues.  For  example, 
the  ABL  contractor  experienced  a  failure  in  some  of  the  system’s  optics 
which  required  it  to  develop  and  procure  new  high  power  optics,  delaying 
the  test  schedule  and  increasing  program  cost.  In  addition,  the  THAAD 
development  contractor  expended  more  funds  than  expected  for  redesigns 
on  the  missile’s  divert  and  attitude  control  system  assembly,  correcting 
issues  with  its  boost  motor,  and  making  changes  on  the  design  of  its 
optical  block — a  safety  system  to  prevent  inadvertent  launches.  Also,  the 
contractor  experienced  cost  overruns  on  extended  testing  and  redesigns 
for  its  prime  power  unit  in  the  radar  portion  of  the  contract. 


Contractors  were  able  to  perform  within  their  fiscal  year  2009  budgeted 
costs  for  three  contracts — the  Aegis  BMD  SM-3  contract  for  a  fourth  lot  of 
27  SM-3  Block  IA  missiles  and  contract  for  another  lot  of  24  SM-3  Block  IA 
missiles,  and  the  BMDS  radars  contract.  The  Aegis  BMD  SM-3  contractor 
attributed  underruns  in  both  of  these  lots  of  Block  LA  missiles  to 
production  efficiencies  since  the  contractor  has  been  building  Aegis  BMD 
SM-3  Block  I  and  LA  missiles  for  nearly  6  years.  The  BMDS  radars 
contractor  improved  cost  performance  during  the  fiscal  year  through 
efficiencies  in  the  software  development  and  systems  engineering. 


Conclusions 


Because  MDA  has  not  established  cost  baselines,  prime  contractor  EVM 
data  provides  one  of  the  only  tools  to  understand  MDA’s  cost  and  schedule 
progress,  particularly  for  purposes  of  external  oversight.  At  present  that 
tool  cannot  be  used  effectively  for  two  major  contractors  because  their 
data  are  not  sufficiently  reliable.  While  MDA  is  taking  action  to  stabilize  its 
programs  and  thereby  improve  the  reliability  of  its  EVM  data,  any 
additional  delays  into  fiscal  year  2011  could  affect  future  fiscal  years’ 
oversight.  Moreover,  until  the  data  are  sufficiently  reliable,  MDA,  GAO  and 
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Congress  lose  the  valuable  insights  into  contractor  performance  that  EVM 
provides,  including  an  understanding  of  significant  drivers  to  performance, 
the  ability  to  forecast  future  cost  and  schedule  performance,  and  the 
ability  to  develop  constructive  corrective  action  plans  based  on  these 
results  to  get  programs  that  have  encountered  problems  back  on  track. 


Recommendation  for 
Executive  Action 


We  recommend  the  Secretary  of  Defense  direct  MDA  to  resolve  prime 
contractor  data  reliability  issues  by  the  beginning  of  fiscal  year  2011  and,  if 
MDA  has  not  resolved  the  data  reliability  problems,  determine  the  barriers 
preventing  resolution  and  provide  a  report  to  Congress  on: 

•  the  steps  MDA  is  taking  to  make  its  contractor  data  sufficiently 
reliable, 

•  how  the  data  reliability  issues  affect  MDA’s  ability  to  provide 
oversight  of  its  contractors,  and 

•  the  effect  these  issues  have  on  MDA’s  ability  to  report  contractor 
progress  to  others,  including  Congress. 


Agency  Comments 
and  Our  Evaluation 


DOD  provided  written  comments  on  a  draft  of  this  report.  These 
comments  are  reprinted  in  appendix  I.  DOD  also  provided  technical 
comments,  which  were  incorporated  as  appropriate. 

DOD  concurred  with  our  recommendation  to  resolve  prime  contractor 
EVM  data  reliability  issues  by  2011;  however,  DOD  stated  that  MDA 
considers  its  fiscal  year  2009  prime  contractor  performance  data  to  be 
reliable.  It  should  be  noted  that,  while  MDA  has  undertaken  extra 
measures  to  gain  insight  into  and  compensate  for  the  program  instability 
effects  on  its  EVM  data,  the  insights  gained  by  MDA  are  not  available  to 
external  organizations  which  depend  on  the  EVM  data  to  analyze  and 
forecast  trends.  Without  the  benefit  of  MDA's  extra  measures  and  added 
reviews,  we  maintain  that  the  prime  contractor  fiscal  year  2009  EVM  data 
are  not  sufficiently  reliable  for  analysis.  Although  we  agree  that  MDA  will 
likely  have  better  insight  into  the  reliability  of  its  contractor  performance 
data  once  it  completes  its  comprehensive  Integrated  Baseline  Review 
process  and  verifies  data  reliability  through  joint  surveillance  of  the 
contractor’s  EVM  system  as  stated  in  the  DOD  response,  we  are  retaining 
the  recommendation  to  ensure  that  these  corrective  steps  are 
implemented  in  time  to  improve  the  reliability  of  the  EVM  data  by  the 
beginning  of  the  next  fiscal  year. 
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We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense,  the 
Director,  MDA,  and  Office  of  Management  and  Budget.  The  report  also  is 
available  at  no  charge  on  the  GAO  Web  site  at  http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  concerning  this  report,  please 
contact  me  at  (202)  512-4841  or  chaplainc@gao.gov.  Contact  points  for  our 
Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found  on  the 
last  page  of  this  report.  GAO  staff  who  made  major  contributions  to  this 
report  are  listed  in  appendix  IV. 


Cristina  T.  Chaplain 
Director 

Acquisition  and  Sourcing  Management 
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List  of  Committees 

The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Daniel  K.  Inouye 
Chairman 

The  Honorable  Thad  Cochran 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Ike  Skelton 
Chairman 

The  Honorable  Howard  P.  McKeon 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Norman  D.  Dicks 
Chairman 

The  Honorable  C.W.  Bill  Young 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Appendix  I:  Comments  from  the  Department 
of  Defense 


ACQUISITION, 
TECHNOLOGY 
AND  LOGISTICS 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3000 


dUL  -1  2010 

Ms.  Christina  T.  Chaplain 
Director,  Acquisition  and  Sourcing  Management 
U.  S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Ms.  Chaplain: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report, 
GAO- 10-676, ‘DEFENSE  ACQUISITIONS:  Missile  Defense  Earned  Value  Management 
Effectiveness  Limited  by  Data  Reliability  Issues,” dated  May  28, 2010  (GAO  Code 
120891). 


The  DoD  concurs  with  the  draft  reports  recommendation.  The  rationale  for  our 
position  is  included  in  the  enclosure.  A  list  of  technical  and  factual  errors  has  been 
submitted  separately  for  your  consideration. 

We  appreciate  the  opportunity  to  comment  on  the  draft  report.  My  point  of 
contact  for  this  effort  is  Mr.  David  Grim,  (703)  697-5385,  david.erim@osd.mil. 


Sincerely, 


/David  G.  Ahem 
/  Director 

(  Portfolio  Systems  Acquisition 


Enclosures: 
As  stated 
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Appendix  I:  Comments  from  the  Department 
of  Defense 


GAO  DRAFT  REPORT-DATED  MAY  28,  2010 
GAO  CODE  120891 /GAO- 10-676 

"DEFENSE  ACQUISITIONS:  Missile  Defense  Earned  Value 
Management  Effectiveness  Limited  by  Data  Reliability  Issues" 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommends  the  Secretary  of  Defense  direct 
MDA  to  resolve  prime  contractor  data  reliability  issues  by  the  beginning  of  fiscal  year 
2011  and,  if  MDA  has  not  resolved  the  data  reliability  problems,  determine  the  barriers 
preventing  resolution  and  provide  a  report  to  Congress  on: 

•The  steps  MDA  is  taking  to  make  its  contractor  data  sufficiently  reliable, 

•How  the  data  reliability  issues  affect  MDA’s  ability  to  provide  oversight  of  its 
contractors,  and 

•The  effect  these  issues  have  on  MDA’s  ability  to  report  contractor  progress  to 
others,  including  Congress. 

DOD  RESPONSE:  Concur.  MDA  considers  the  prime  contractor  data  to  be 
sufficiently  reliable  to  assess  fiscal  year  2009  performance.  MDA  expects  data  reliability 
issues  to  be  resolved  by  the  beginning  of  fiscal  year  201 1  and  that  accurate  contractor 
progress  will  be  reported  to  outside  entities,  including  Congress.  MDA  will  verify  data 
reliability  through  joint  surveillance  of  the  contractor's  Earned  value  Management  System 
and  a  comprehensive  Integrated  Baseline  Review  (IBR)  process.  All  IBRs  will  be 
conducted  by  a  joint  team  consisting  of  the  program  office,  the  contractor,  and  Defense 
Contract  Management  Agency  (DCMA). 
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Appendix  II:  BMDS  Prime  Contractor  Fiscal 
Year  2009  Cost  and  Schedule  Performance 


To  determine  if  they  are  executing  the  work  planned  within  the  funds  and 
time  budgeted,  each  prime  contractor  provides  monthly  reports  detailing 
cost  and  schedule  performance.  The  contractor  tracks  earned  value 
management  (EVM)  by  making  comparisons  that  inform  the  program  as  to 
whether  the  contractor  is  completing  work  at  the  cost  budgeted  and 
whether  the  work  scheduled  is  being  completed  on  time  and  then  reports 
this  information  on  Contract  Performance  Reports.1  For  example,  if  the 
contractor  was  able  to  complete  more  work  than  scheduled  and  for  less 
cost  than  budgeted,  the  contractor  reports  a  positive  schedule  and  cost 
variance,  or  “underrun”.  Alternatively,  if  the  contractor  was  not  able  to 
complete  the  work  in  the  scheduled  time  period  and  spent  more  than 
budgeted,  the  contractor  reports  both  a  negative  schedule  and  cost 
variance,  or  “overrun”.  The  results  can  also  be  mixed  by,  for  example, 
completing  the  work  ahead  of  schedule  (a  positive  schedule  variance)  but 
spending  more  than  budgeted  to  do  so  (a  negative  cost  variance). 

We  provide  two  kind  of  variances  in  our  individual  contract  assessments 
pertaining  to  overruns  or  underruns  either  cumulatively  over  the  life  of  the 
contract  or  during  the  fiscal  year.  Cumulative  variances  are  the  overruns 
or  underruns  the  contractor  has  earned  since  the  contract  began.  In  order 
to  calculate  fiscal  year  variances,  we  determined  the  contractor’s 
cumulative  variances  at  the  end  of  September  2008  and  subtracted  them 
from  the  cumulative  variances  at  the  end  of  September  2009.  Fiscal  year 
2009  variances  give  us  an  idea  of  the  contractor’s  performance  trends 
during  the  fiscal  year.  A  contractor  may  have  cumulative  overruns  but 
underrun  its  fiscal  year  budgeted  cost  or  schedule  by  improving  its  cost 
performance  over  the  course  of  the  fiscal  year. 

In  our  graphs,  positive  fiscal  year  variances  (underrunning  cost  or 
schedule)  are  indicated  by  increasing  performance  trend  lines  and 
negative  fiscal  year  variances  (overrunning  cost  or  schedule)  are  shown  by 
decreasing  performance  trend  lines.  In  our  notional  example  in  Figure  1, 
the  positive  slope  of  the  cost  variances  line  indicates  that  the  contractor  is 
underrunning  fiscal  year  budgeted  cost.  Specifically,  the  contractor  began 
the  fiscal  year  with  a  negative  cumulative  cost  variance  of  $7.0  million  but 


1  Earned  value  management  is  a  program  management  tool  that  integrates  the  technical, 
cost,  and  schedule  parameters  of  a  contract.  During  the  planning  phase,  an  integrated 
baseline  is  developed  by  time-phasing  budget  resources  for  defined  work.  As  work  is 
performed  and  measured  against  the  baseline,  the  corresponding  budget  value  is  “earned.” 
Using  this  earned  value  metric,  cost  and  schedule  variances  can  be  determined  and 
analyzed. 
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Appendix  II:  BMDS  Prime  Contractor  Fiscal 
Year  2009  Cost  and  Schedule  Performance 


ended  the  fiscal  year  with  a  negative  cumulative  cost  variance  of  $1.0 
million.  That  means  that  the  contractor  underran  its  fiscal  year  budgeted 
costs  by  $6.0  million  and  therefore  has  a  positive  $6.0  million  fiscal  year 
cost  variance.  Alternately,  the  cumulative  schedule  variance  is  decreasing 
during  the  fiscal  year  indicating  that  the  contractor  was  unable  to 
accomplish  planned  fiscal  year  work  and  therefore  has  a  negative  fiscal 
year  schedule  variance.  In  this  case,  the  schedule  performance  declined 
during  the  fiscal  year  from  $5.0  million  down  to  $2.0  million.  Therefore, 
the  contractor  was  unable  to  accomplish  $3.0  million  worth  of  work 
planned  during  the  fiscal  year. 


Figure  1 :  Depiction  of  Notional  Contractor  Cumulative  Cost  and  Schedule  Performance 


Dollars  In  millions 


2008 


2009 


Month 


Cumulative  cost  variance 
■  ■■■  Cumulative  schedule  variance 
Source:  GAO. 


The  individual  points  on  Figure  1  also  show  the  cumulative  performance 
over  the  entire  contract  up  to  each  month.  Points  in  a  month  that  are 
above  $0  million  represent  a  positive  cumulative  variance  (underrunning 
cost  or  schedule)  and  points  below  $0  million  represent  a  negative 
cumulative  variance  (overrunning  cost  or  schedule).  In  our  notional 
example,  the  contractor  ended  the  fiscal  year  with  a  negative  cumulative 
cost  variance  of  $1.0  million.  This  means  that  since  the  contract’s 
inception,  the  contractor  is  overrunning  its  budgeted  cost  by  $1.0  million. 
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Alternately,  the  contractor  ended  the  fiscal  year  with  a  positive  cumulative 
schedule  variance  of  $2.0  million.  That  means  that  over  the  life  of  the 
contract,  the  contractor  has  been  able  to  accomplish  $2.0  million  more 
worth  of  work  than  originally  planned. 

Besides  reporting  cost  and  schedule  variances,  we  also  used  contract 
performance  report  data  to  base  projections  of  likely  overrun  or  underrun 
of  each  prime  contractor’s  budgeted  cost  at  completion.  Our  projections  of 
overruns  or  underruns  to  the  budgeted  cost  at  completion  are  based  on 
the  assumption  that  the  contractor  will  continue  to  perform  in  the  future 
as  it  has  in  the  past.  Our  projections  are  based  on  the  current  budgeted 
costs  at  completion  for  each  contract  we  assessed,  which  represents  the 
total  planned  value  of  the  contract  as-of  September  2009.  However,  the 
budgeted  costs  at  completion,  in  some  cases,  have  grown  significantly 
over  time.  For  example,  the  Airborne  Laser  (ABL)  contractor  reported 
budgeted  costs  at  completion  totaling  about  $724  million  in  1997,  but  that 
cost  has  since  grown  to  about  $3.7  billion. 

Our  assessment  only  reveals  the  overrun  or  underrun  since  the  latest 
adjustment  to  the  budget  at  completion.  It  does  not  capture,  as  cost 
growth,  the  difference  between  the  original  and  current  budgeted  costs  at 
completion.  As  a  result,  comparing  the  underruns  or  overruns  for  Missile 
Defense  Agency  (MDA)  programs  with  cost  growth  on  other  major  defense 
acquisition  programs  is  not  appropriate  because  MDA  has  not  developed 
the  full  scope  of  work  and  total  cost  baselines  that  other  major  defense 
acquisition  programs  have. 


Aegis  BMD 
Contractors 
Experienced  Mixed 
Performance  during 
the  Fiscal  Year 


The  Aegis  Ballistic  Missile  Defense  (BMD)  program  employs  two  prime 
contractors  for  its  two  main  components — Lockheed  Martin  Mission 
Systems  and  Sensors  for  the  Aegis  BMD  Weapon  System  and  Raytheon  for 
the  Aegis  BMD  Standard  Missile-3  (SM-3).  During  fiscal  year  2009,  the 
Aegis  BMD  SM-3  Block  LA  and  IB  missile  technology  development  and 
engineering  contract  experienced  declining  cost  and  schedule 
performance,  the  Aegis  BMD  SM-3  contract  for  a  fourth  lot  of  27  Block  IA 
missiles  had  increasing  cost  and  schedule  performance,  and  the  Aegis 
Weapon  System  and  Aegis  BMD  SM-3  contractor  for  another  lot  of  24  SM-3 
Block  LA  missiles  experienced  mixed  performance. 


Aegis  BMD  Weapon  Although  the  Aegis  Weapon  System  contractor  overran  fiscal  year  2009 

System  budgeted  costs  by  $0.2  million,  it  was  able  to  accomplish  $1.7  million  more 

worth  of  work  than  originally  anticipated.  The  fiscal  year  2009  cost 
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overrun  is  attributed  to  unplanned  complexity  associated  with  developing 
radar  software.  During  the  fiscal  year,  the  decline  in  cost  performance  and 
subsequent  recovery  is  partially  attributed  to  annual  technical  instruction 
baseline  updates.  These  baseline  updates  occur  over  the  course  of  a  sixty 
day  period  during  which  varying  performance  data  occurs.  At  the  end  of 
this  period,  there  is  a  jump  in  performance  as  the  contractor  earns  two 
months  worth  of  performance.  Some  of  the  cost  savings  from  April 
through  September  2009  are  the  result  of  a  planned  flight  test  being 
cancelled  during  the  fiscal  year  and  the  contractor  not  spending  intended 
funds  on  pre-flight  test,  flight  test,  and  post-flight  test  activities.  The 
favorable  schedule  variance  was  driven  by  completion  of  some  technical 
instruction  efforts.  Figure  2  shows  cumulative  variances  at  the  beginning 
of  fiscal  year  2009  along  with  a  depiction  of  the  contractor’s  cost  and 
schedule  performance  throughout  the  fiscal  year. 


Figure  2:  Aegis  BMD  Weapon  System  Cumulative  Cost  and  Schedule  Performance 
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■  ■■■  Cumulative  schedule  variance 
Sources:  Contractor  (data):  GAO  (presentation). 


Considering  prior  performance  on  the  Aegis  Weapon  System  contract 
since  it  began  performance  in  October  2003,  the  contractor  is  $0.2  million 
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over  budget  and  has  been  unable  to  accomplish  $6.7  million  worth  of 
work.  The  small  negative  cost  variance  was  driven  primarily  by  radar 
software  development  issues,  including  a  significant  redesign  not  included 
in  the  original  baseline.  In  addition,  the  engineering  test  and  evaluation 
portion  of  the  radar  software  is  experiencing  an  increase  in  the  lines  of 
code  that  also  accounts  for  some  of  the  budget  overrun.  The  unfavorable 
$6.7  million  in  schedule  variances  are  attributed  to  the  engineering  test 
and  evaluation  portion  of  the  radar  software  for  which  builds  and 
capabilities  are  being  delivered  later  than  originally  planned.  If  the 
contractor  continues  to  perform  as  it  did  through  September  2009,  our 
analysis  projects  that  at  completion  in  June  2010,  the  work  under  the 
contract  could  cost  about  $0.2  million  more  than  the  budgeted  cost  of  $1.5 
billion. 


Aegis  BMD  SM-3  for  27  The  Aegis  BMD  SM-3  contractor  for  a  fourth  lot  of  27  Block  IA  missiles 

Block  IA  Missiles  underran  its  budgeted  fiscal  year  2009  cost  and  schedule  by  $0.5  million 

and  $5.8  million  respectively.  The  program  attributed  its  cost  and  schedule 
underruns  to  efficiencies  in  producing  Aegis  BMD  SM-3  Block  I  and  IA 
missiles  since  the  contractor  has  been  building  these  missiles  for  nearly  6 
years.  Additionally,  the  program  reported  that  the  contract  incentivizes  the 
contractor  to  deliver  missiles  ahead  of  schedule  for  maximum  incentive 
fee  which  further  encouraged  the  contractor  to  accomplish  $5.8  million 
more  worth  of  work  then  originally  planned  during  the  fiscal  year.  See 
figure  3  for  an  illustration  of  cumulative  cost  and  schedule  variances 
during  the  course  of  the  fiscal  year. 
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Figure  3:  Aegis  BMD  SM-3  Contract  for  27  Block  IA  Missiles  Cumulative  Cost  and  Schedule  Performance 
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Considering  prior  years’  performance  since  the  contract  began  in  May 
2007,  the  contractor  is  performing  under  budgeted  cost  with  a  favorable 
cumulative  cost  variance  of  $3.9  million  but  is  behind  schedule  on  $1.3 
million  worth  of  work.  The  cost  underruns  are  primarily  driven  by 
implemented  efficiencies,  material  transfers,  and  program  management 
adjustments  within  the  solid  divert  and  attitude  control  system;  a  decrease 
in  rework  and  more  efficiencies  realized  with  the  seeker;  and  underruns  in 
engineering  efforts  associated  with  the  third  stage  rocket  motor.  The  $1.3 
million  in  schedule  overruns  are  attributed  to  late  delivery  of  parts  as  the 
result  of  some  equipment  failures.  If  the  contractor  continues  to  perform 
as  it  did  through  September  2009,  our  analysis  projects  that  at  completion 
in  December  2011,  the  work  under  the  contract  could  cost  about  $5.2 
million  less  than  the  budgeted  cost  of  $233.8  million. 
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Aegis  BMD  SM-3  for  24  As  of  September  2009,  the  Aegis  BMD  SM-3  contractor  for  another  lot  of  24 

Block  IA  Missiles  Block  LA  missiles  had  underrun  its  fiscal  year  budget  by  $4.2  million  and 

was  behind  in  completing  $3.7  million  worth  of  work.  The  contractor 
attributes  its  cost  underrun  to  efficiencies  in  program  management  and 
systems  engineering  because  of  its  experience  in  building  SM-3  Block  I 
and  LA  missiles.  The  $3.7  million  in  schedule  overruns  resulted  from  the 
contractor  planning  the  baseline  to  a  more  aggressive  schedule  than  the 
contractual  missile  delivery  schedule  requires.  The  contractor  plans  in  this 
way  because  it  is  incentivized  to  deliver  missiles  2  months  ahead  of 
schedule.  As  a  result,  negative  schedule  variances  have  occurred  as  the 
contractor  is  pushing  to  deliver  missiles  early.  Figure  4  shows  both  cost 
and  schedule  trends  during  fiscal  year  2009. 


Figure  4:  Aegis  BMD  SM-3  Contract  for  24  Block  IA  Missiles  Cumulative  Cost  and  Schedule  Performance 

Dollars  in  millions 
8 


6 

4 

2 

0 

-2 

-4 

-6 


Sept.  Oct.  Nov.  Dec. 

2008 
Month 


Jan.  Feb.  Mar.  Apr.  May  June  July  Aug.  Sept. 

2009 


38.7  percent  of  contract  complete 


Cumulative  cost  variance 
■  ■■■  Cumulative  schedule  variance 
Sources:  Contractor  (data):  GAO  (presentation). 


Cumulatively,  since  the  contract  began  in  February  2008,  the  contractor  is 
underrunning  its  contract’s  budgeted  cost  by  $1.4  million  but  is  behind  on 
$2.1  million  worth  of  work.  The  contractor  attributes  the  cost  underrun  to 
labor  efficiencies  and  reduced  manpower  within  the  seeker  design  as  well 
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as  a  slower-than-planned  ramp-up  of  some  engineering  efforts.  The 
schedule  delays  are  mainly  driven  by  non-delivery  of  parts  for  the  first 
stage  rocket  motor  and  late  deliveries  of  parts  associated  with  the  third 
stage  rocket  motor.  If  the  contractor  continues  to  perform  as  it  did 
through  September  2009,  our  analysis  projects  that  at  completion  in 
December  2011,  the  work  under  the  contract  could  cost  from  $15.3  million 
less  to  $1.9  million  more  than  the  budgeted  cost  of  $192.6  million. 


Aegis  BMD  SM-3  Block  IA 
and  IB  Technical 
Development  and 
Engineering 


For  the  majority  of  the  fiscal  year,  the  Aegis  BMD  SM-3  Block  IA  and  IB 
Technical  Development  and  Engineering  contractor  experienced  a 
negative  downward  trend  in  cost  and  schedule  performance.  The  program 
attributes  its  fiscal  year  cost  overrun  of  $44.6  million  to  engineering 
development  on  its  Aegis  BMD  SM-3  Block  IB  throttleable  divert  and 
attitude  control  system  being  more  difficult  than  planned.  The  $29.4 
million  of  unaccomplished  work  during  the  fiscal  year  was  due  to  late 
receipt  of  materials  that  drove  delays  in  some  of  the  hardware  testing.  See 
figure  5  for  trends  in  the  contractor’s  cost  and  schedule  performance 
during  the  fiscal  year. 
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Figure  5:  Aegis  BMD  SM-3  Block  IA  and  IB  Technical  Development  and  Engineering  Cumulative  Cost  and  Schedule 
Performance 
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Cumulatively,  since  the  contract  began  in  December  2007,  the  program 
also  has  unfavorable  cost  and  schedule  variances  of  $51.2  million  and 
$40.0  million,  respectively.  Drivers  of  the  $51.2  million  in  cost  overruns  are 
throttleable  divert  and  attitude  control  system  engineering  and  hardware 
major  submaterial  price  increases  in  support  of  design  reviews  and 
demonstration  unit.  In  addition,  quality  issues  added  to  cost  overruns  as 
the  contractor  experienced  unanticipated  design  changes  to  the  nozzle 
resulting  from  foreign  object  debris  issues.  The  $40.0  million  worth  of 
work  that  the  contractor  was  unable  to  achieve  was  driven  by  several 
issues,  including  late  receipt  of  hardware  and  late  production-level 
drawings.  In  addition,  delays  in  testing  for  attitude  control  system 
thrusters  and  a  quality  issue  that  led  to  the  contractor  receiving 
nonconforming  hardware  also  contributed  to  unaccomplished  work.  If  the 
contractor  continues  to  perform  as  it  did  through  September  2009,  our 
analysis  projects  that  at  completion  in  December  2010,  the  work  under  the 
contract  could  cost  from  $94.0  million  to  $194.8  million  more  than  the 
budgeted  cost  of  $588.9  million. 
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ABL  Cost  and 
Schedule 

Performance  Declined 
during  Fiscal  Year 
2009  ' 


The  ABL  contractor,  Boeing,  experienced  cost  growth  and  schedule  delays 
throughout  the  fiscal  year.  The  contractor  overran  budgeted  fiscal  year 
2009  cost  and  schedule  by  $10.2  million  and  $14.9  million  respectively.  The 
major  drivers  of  fiscal  year  negative  variances  were  technical  issues  and 
the  addition  of  some  testing  that  was  not  originally  anticipated.  For 
example,  a  fire  suppression  system  failed  to  meet  performance 
requirements  for  the  laser  flight  test  which  limited  the  scope  of  the  testing, 
added  an  unscheduled  ground  test  and  flight  tests  to  ensure  that  the 
system  worked  properly,  and  increased  costs.  In  addition,  the  contractor 
experienced  a  failure  in  some  of  the  system’s  optics  which  required  it  to 
develop  and  procure  new  high-power  optics  and  ultimately  delayed  the 
test  schedule  and  increased  program  cost.  Lastly,  because  of  issues 
discovered  during  beam  control/fire  control  flights,  the  program  scheduled 
additional  unplanned  beam  control  flights  to  accomplish  the  necessary 
objectives.  The  contractor  experienced  a  continuing  cost  and  schedule 
performance  decline,  as  seen  in  figure  6. 
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Figure  6:  ABL  Cumulative  Cost  and  Schedule  Performance 
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The  contractor’s  cumulative  cost  variance  is  over  budget  by  $95.0  million 
and  behind  schedule  by  $38.5  million  from  when  the  contract  began  in 
November  1997.  The  program  attributes  these  variances  to  optics  issues 
that  have  affected  delivery  and  installation  and  caused  test  program 
delays.  If  the  contractor  continues  to  perform  as  it  did  through  September 
2009,  our  analysis  projects  that  at  completion  in  February  2010,  the  work 
under  the  contract  could  cost  from  $98.0  million  to  $116.8  million  more 
than  the  budgeted  cost  of  $3.7  billion.2 


2  Since  September  2009,  the  period  of  performance  for  the  contract  was  extended  to  August 
2010. 
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C2BMC  Overrunning 
Cumulative  Cost  and 
Schedule 


The  Command  and  Control,  Battle  Management,  and  Communications 
(C2BMC)  contractor,  Lockheed  Martin  Information  Systems  &  Global 
Services,  is  currently  overrunning  budgeted  costs  for  the  agreement  since 
it  began  performance  in  February  2002  by  $29.5  million  and  has  a 
cumulative  schedule  variance  of  $4.2  million.3  According  to  program 
officials,  the  main  drivers  of  the  cumulative  variances  are  associated  with 
the  Part  4  and  Part  5  portions  of  the  agreement.  The  Part  4  effort,  which 
began  in  January  2006  and  finished  December  2007,  was  for  the 
completion  of  several  spiral  capabilities,  the  upgrade  for  spiral  suites,  and 
implementation  of  initial  global  engagement  capabilities  at  its  operations 
center.  The  Part  5  effort,  which  began  in  January  2008  and  is  still  ongoing, 
covers  operations  and  sustainment  support  for  fielded  C2BMC;  deliveries 
of  spiral  hardware,  software,  and  communications;  and  initiated 
development  of  initial  global  engagement  capabilities.  MDA  and  the 
contractor  anticipate  being  able  to  cover  cost  overruns  on  the  agreement 
with  the  nearly  $39  million  in  management  reserve  set  aside  by  the 
contractor. 


Part  5  accounts  for  nearly  $10.4  million  of  the  $29.5  million  in  negative 
cumulative  cost  variances.  These  budgeted  cost  overruns  are  driven  by 
increased  technical  complexity  of  Spiral  6.4  development,  and  more 
support  needed  than  planned  to  address  requests  from  the  warfighter  for 
software  modifications.  The  $4.2  million  of  unaccomplished  work  on  the 
agreement  is  driven  by  efforts  in  the  Part  5  portion  of  the  agreement, 
including  delays  in  system  level  tests,  late  completion  of  C2BMC  interface 
control  document  updates,  and  unexpected  complexity  of  algorithm 
development  and  network  design.  See  figure  7  for  an  illustration  of 
cumulative  cost  and  schedule  performance  during  fiscal  year  2009. 


3  The  C2BMC.  element  operates  under  an  Other  Transaction  Agreement  with  cost 
reimbursement  aspects.  These  types  of  agreements  are  not  always  subject  to  procurement 
laws  and  regulations  meant  to  safeguard  the  government.  MDA  chose  the  Other 
Transaction  Agreement  to  facilitate  a  collaborative  relationship  between  industry, 
government,  federally  funded  research  and  development  centers,  and  university  research 
centers.  For  the  purposes  of  this  report,  we  have  included  this  agreement  in  our  analysis  of 
BMDS  contracts. 
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Figure  7:  C2BMC  Cumulative  Cost  and  Schedule  Performance 
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The  contractor  overran  its  fiscal  year  2009  budgeted  cost  by  $5.2  million 
but  is  $2.9  million  ahead  of  schedule.  The  drivers  of  the  unfavorable  fiscal 
year  cost  variance  of  $5.2  million  are  complexities  associated  with  Spiral 
6.4  development,  additional  design  excursions,  and  additional  costs  to 
address  system  modifications  requested  by  the  warfighter.  The  contractor 
achieved  a  favorable  fiscal  year  schedule  variance  largely  because  of  gains 
in  the  month  of  September  2009.  During  this  month,  the  contractor 
performed  a  replan  of  its  work  content  and  a  future  spiral’s  scope  was 
removed  from  the  Part.  This  replan  eliminated  approximately  $10  million 
in  schedule  variances  for  labor  and  materials  because  the  work  was  no 
longer  to  be  performed.  If  the  contractor  continues  to  perform  as  it  did 
through  September  2009,  our  analysis  projects  that  at  completion  in 
December  2011,  the  work  under  the  agreement  could  cost  from  $26.5 
million  to  $33.1  million  more  than  the  budgeted  cost  of  $1.0  billion. 
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Sensors  Contractor 
Experienced  Mixed 
Performance  during 
the  Fiscal  Year 


This  year  we  are  reporting  on  three  contracts  under  the  Sensors 
program — the  Ballistic  Missile  Defense  System  (BMDS)  Radars  contract 
on  which  we  have  reported  in  prior  years,  the  Terminal  High  Altitude  Area 
Defense  (THAAD)  fire  unit  radar  #7  contract,  and  the  Thule  radar  contract. 
During  fiscal  year  2009,  the  Sensors’  contractor,  Raytheon,  experienced 
declining  cost  and  schedule  performance  on  the  Thule  radar  and  Army 
Navy/Transportable  Radar  Surveillance — Model  2  (AN/TPY-2)  radar  #7 
contracts,  but  had  favorable  cost  and  schedule  performance  on  the  BMDS 
Radars  contract. 


BMDS  Radars  Throughout  fiscal  year  2009,  the  BMDS  Radars  contractor  exhibited 

improved  cost  and  schedule  performance.  The  contractor  was  able  to 
perform  $5.8  million  under  budgeted  cost  and  $3.5  million  ahead  of 
schedule  for  the  fiscal  year.  The  drivers  of  the  contractor’s  improved  cost 
performance  are  efficiencies  in  the  software  development  and  systems 
engineering.  The  contractor  reports  that  the  improved  schedule 
performance  is  due  to  software  schedule  improvement  as  well  as 
completion  of  manufacturing  and  integration  testing  on  one  of  the  radars. 
The  variances,  depicted  in  figure  8,  represent  the  BMDS  Radars 
contractor’s  cumulative  cost  and  schedule  performance  over  fiscal  year 
2009. 
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Figure  8:  BMDS  Radars  Cumulative  Cost  and  Schedule  Performance 
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Since  the  contract  began  in  March  2003,  the  BMDS  Radars  contractor  is 
under  budget  by  $27.8  million  but  is  behind  on  accomplishing  $6.1  million 
worth  of  work.  The  favorable  cost  variance  of  $27.8  million  is  driven  by 
the  use  of  less  manpower  than  planned  and  the  benefit  of  lessons  learned 
from  previous  radar  software  builds.  The  unfavorable  $6.1  million  of 
unaccomplished  work  was  driven  by  the  late  start  on  restructuring  the 
latest  software  release  and  rework  and  subcomponent  delays  with  one  of 
the  radars.  If  the  contractor  continues  to  perform  as  it  did  through 
September  2009,  our  analysis  projects  that  at  completion  in  August  2010, 
the  work  under  the  contract  could  cost  from  $31.3  million  to  $43.0  million 
less  than  the  budgeted  cost  of  $1.2  billion. 


AN/TPY-2  #7  Radar  The  AN/TPY-2  radar  #7  contractor  experienced  unfavorable  fiscal  year 

2009  cost  and  schedule  variances  of  $4.3  million  and  $15.2  million, 
respectively.  As  of  September  2009,  the  AN/TPY-2  radar  #7  contract  had 
overrun  its  budgeted  cost  by  $1.9  million  but  was  ahead  in  completing  $9.0 
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million  worth  of  work.  Contributors  to  the  cumulative  cost  overruns 
included  supplier  quality  issues  that  required  an  increase  in  supplier 
quality  support  that  was  not  in  the  original  baseline.  In  addition,  the 
program’s  prime  power  unit  purchase  orders  were  over  budgeted  cost 
because  the  budgeted  cost  for  four  of  the  prime  power  units  was 
prematurely  established  before  the  design  of  the  first  prime  power  unit 
was  finalized.  These  delays  caused  some  uncertainty  in  the  final 
production  costs  until  the  design  was  finalized.  As  of  August  2009,  the 
contractor  was  working  to  develop  a  cost  model  and  establish  a  true  unit 
cost  price  per  prime  power  unit.  Trends  in  cost  and  schedule  performance 
during  the  fiscal  year  are  depicted  in  figure  9. 


Figure  9:  AN/TPY-2  Radar  #7  Cumulative  Cost  and  Schedule  Performance 
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Cumulatively,  since  the  contract  began  in  February  2007,  the  AN/TPY-2 
Radar  #7  contractor  has  completed  $9.0  million  worth  of  work  ahead  of 
schedule  on  this  contract  by  executing  work  ahead  of  the  contract 
baseline  plan  in  some  areas,  including  obtaining  materials  for  equipment 
supporting  radar  operation.  If  the  contractor  continues  to  perform  as  it  did 


Page  36 


GAO-10-676  Missile  Defense  Contractor  Performance 


Appendix  II:  BMDS  Prime  Contractor  Fiscal 

Year  2009  Cost  and  Schedule  Performance 

through  September  2009,  our  analysis  projects  that  at  completion  in  April 
2010,  the  work  under  the  contract  could  cost  from  $0.3  million  less  to 
$36.9  million  more  than  the  budgeted  cost  of  $172.5  million. 

Thule  Radar 

The  Thule  radar  contractor  overran  fiscal  year  2009  budgeted  costs  by  $0.4 
million  and  was  unable  to  accomplish  $0.8  million  worth  of  work.  The 
contractor  attributes  the  cost  overruns  to  exceeding  planned  engineering 
efforts  in  order  to  proactively  work  on  issues  prior  to  equipment  delivery 
and  ship  readiness.  The  unfavorable  schedule  performance  is  due  to  the 
contractor  expending  some  if  its  positive  schedule  variance  in  2008  and 
from  being  behind  schedule  on  the  implementation  of  information 
assurance  requirements.  Figure  10  shows  cumulative  variances  at  the 
beginning  of  fiscal  year  2009  along  with  a  depiction  of  the  contractor’s 
cost  and  schedule  performance  throughout  the  fiscal  year. 

Figure  10:  Thule  Radar  Cumulative  Cost  and  Schedule  Performance 
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The  Thule  radar  contractor,  since  it  began  performance  in  April  2006,  is 
underrunning  budgeted  costs  by  $2.6  million  and  overrunning  schedule  by 
$0.2  million.  Underruns  in  hardware,  manufacturing,  and  facility  design, 
construction,  and  installation  drove  the  $2.5  million  in  cost  underruns.  If 
the  contractor  continues  to  perform  as  it  did  through  September  2009,  our 
analysis  projects  that  at  completion  in  September  2010,  the  work  under 
the  contract  could  cost  from  $1.4  million  to  $2.8  million  less  than  the 
budgeted  cost  of  $101.9  million. 


STSS  Maintained 
Schedule 
Performance,  but 
Cost  Performance 
Continued  to  Decline 
during  the  Fiscal  Year 


During  fiscal  year  2009,  the  Space  Tracking  and  Surveillance  System 
(STSS)  contractor,  Northrop  Grumman,  was  able  to  accomplish  $0.1 
million  more  worth  of  work  than  originally  anticipated,  but  overran 
budgeted  costs  by  $72.6  million.  The  contractor  reports  that  the  favorable 
schedule  variances  are  due  to  completed  space  vehicle  1  and  2  shipment, 
setups  and  validations,  and  launch.  In  addition,  the  contractor  overran 
budgeted  fiscal  year  costs  because  of  additional  support  required  to 
support  launch  operations  including  addressing  hardware  anomalies, 
payload  integration,  procedure  development,  and  launch  site  activities. 
Additional  support  was  also  required  to  support  the  delays  to  the  launch 
date  beyond  the  original  plan.  See  figure  11  for  an  illustration  of  the 
cumulative  cost  and  schedule  variances  during  fiscal  year  2009. 
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Figure  11 :  STSS  Cumulative  Cost  and  Schedule  Performance 
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Note:  The  STSS  contract  includes  line  items  for  work  that  do  not  necessarily  apply  to  the  satellites 
that  were  launched  in  the  fourth  quarter  of  fiscal  year  2009.  Removing  these  line  items  from  our 
analysis,  the  program's  contractwould  be  considered  84.1  percent  complete. 

Despite  the  small  gains  in  schedule  variances  during  the  fiscal  year,  the 
contractor  maintains  cumulative  negative  cost  and  schedule  variances  of 
$391.8  million  and  $17.7  million  respectively  from  the  contract’s  inception 
in  August  2002.  Drivers  of  the  $391.8  million  in  contract  cost  overruns  are 
for  labor  resources  exceeding  planned  levels  and  unanticipated  difficulties 
related  to  space  vehicle  environment  testing,  hardware  failures  and 
anomalies,  and  program  schedule  extension.  In  addition,  space  vehicle-1 
testing,  rework,  hardware  issues,  and  sensor  testing  anomaly  resolution  as 
well  as  space  vehicle-2  anomalies  and  testing  have  also  contributed  to  the 
unfavorable  cost  variances.  System  test  and  operations  and  program 
management  experienced  cost  overruns  because  of  launch  date  schedule 
extensions.  Lastly,  ground  labor  resources  exceeded  planned  levels 
because  of  the  unanticipated  need  for  a  new  ground  software  build  and 
ground  acceptance  and  verification  report  activities.  The  contractor  has 
been  unable  to  accomplish  $17.7  million  worth  of  work  on  the  contract 
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because  of  launch  schedule  delays,  delays  in  verification  of  system 
requirements  caused  by  late  space  segment  deliveries,  and  tasks  slipping 
in  response  to  fiscal  year  2009  funding  reductions.  If  the  contractor 
continues  to  perform  as  it  did  through  September  2009,  our  analysis 
projects  that  at  completion  in  September  2010,  the  work  under  the 
contract  could  cost  from  $620.9  million  to  $1.6  billion  more  than  the 
budgeted  cost  of  $1.6  billion. 


THAAD  Development 
Contract  Overran 
Cost  and  Schedule 
While  THAAD  Fire 
Unit  Fielding 
Production  Contract 
Experienced 
Underruns 


This  year  we  report  on  two  THAAD  contracts — the  development  contract 
and  the  fire  unit  fielding  production  contract.  As  the  contractor  for  both  of 
these  contracts,  Lockheed  Martin  Space  Systems  Company  was 
overrunning  budgeted  cost  and  schedule  on  the  THAAD  development 
contract  but  remained  under  cost  and  ahead  of  schedule  on  the  THAAD 
fire  unit  fielding  production  contract. 


THAAD  Development  During  fiscal  year  2009,  the  THAAD  development  contractor  overran  its 

budgeted  cost  by  $33.1  million  but  was  ahead  on  completing  $7.4  million 
worth  of  work.4  The  fiscal  year  cost  overruns  are  mainly  in  the  missile, 
launcher,  and  radar  portions  of  the  contract.  The  missile  experienced 
overruns  on  divert  and  attitude  control  system  assembly  redesigns, 
correcting  issues  with  its  boost  motor,  and  making  changes  on  the  design 
of  its  optical  block — a  safety  system  to  prevent  inadvertent  launches.  The 
contractor  spent  more  than  expected  during  the  fiscal  year  on  the 
launcher  portion  of  the  contract,  investing  in  labor  and  overtime  to 
recover  schedule.  Lastly,  the  prime  power  unit  in  the  radar  portion  of  the 
contract  required  extended  testing  and  redesign,  which  also  contributed  to 
fiscal  year  costs. 


4  Earned  value  data  for  the  THAAD  development  contract  is  reported  under  two  contract 
line  item  numbers — 1  and  10.  We  report  only  the  contractor’s  cost  and  schedule 
performance  for  contract  line  item  number  1  because  it  represents  the  majority  of  the  total 
work  performed  under  the  contract.  Contract  line  item  number  10  provides  for  Patriot 
Common  Launcher  initiatives  funded  by  the  Army’s  Lower  Tier  Program  Office. 
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Despite  fiscal  year  cost  overruns,  the  contractor  was  able  to  accomplish 
$7.4  million  more  worth  of  work  than  originally  anticipated  also  in  the 
missile  and  launcher  portions  of  the  contract.  The  schedule  variance 
improved  in  the  missile  portion  because  of  completion  of  missile 
qualification  work.  The  contractor  was  also  able  to  complete  software 
activities  and  resolve  hardware  design  and  qualification  issues  in  the 
launcher.  See  figure  12  for  trends  in  the  contractor’s  cost  and  schedule 
performance  during  the  fiscal  year. 


Figure  12:  THAAD  Development  Cumulative  Cost  and  Schedule  Performance 
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Sources:  Contractor  (data);  GAO  (presentation). 


Although  the  contractor  made  some  schedule  gains  during  the  fiscal  year, 
overall  the  contractor  since  it  began  performance  in  June  2000  is  behind 
on  $9.1  million  worth  of  work.  The  radar’s  portion  of  unfavorable  schedule 
variance  is  driven  by  delays  to  THAAD  flight  test  missions  during  fiscal 
year  2009.  In  addition,  the  fire  control’s  software  qualification  testing  had 
to  be  extended  because  of  the  number  of  software  changes  and  because 
the  welding  on  the  fire  control  power  distribution  unit’s  chassis  failed 
weld  inspection  and  was  subsequently  unusable  which  contributed  to  the 
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unfavorable  schedule  variance.  The  launcher  experienced  design  delays 
and  quality  issues  that  led  to  nonconformances  in  delivered  hardware. 

This  hardware  subsequently  required  investigation  and  rework,  which  also 
added  unexpected  work  to  the  schedule.  Lastly,  the  program  was  unable 
to  accomplish  work  in  the  missile  component’s  flight  sequencing  assembly 
component  because  qualification  tests  were  delayed  due  to  failures  with 
the  optical  block  switch. 

The  unfavorable  fiscal  year  cost  variances  added  to  the  overall  cost 
overruns  of  $261.9  million.  The  contractor  attributes  overruns  to  the 
missile,  launcher,  and  radar  portions  of  the  contract.  The  missile’s 
unfavorable  cost  variance  is  driven  by  unexpected  costs  in  electrical 
subsystems,  propulsion,  and  divert  and  attitude  control  systems.  Also 
contributing  are  issues  associated  with  the  optical  block,  range  safety, 
communications  systems,  and  boost  motors.  The  launcher  has 
experienced  cost  growth  because  of  inefficiencies  that  occurred  during 
hardware  design,  integration  difficulties,  quality  issues  leading  to  delivered 
hardware  nonconformances,  and  ongoing  software  costs  being  higher  than 
planned  because  of  rework  of  software  to  correct  testing  anomalies.  These 
problems  resulted  in  schedule  delays  and  higher  labor  costs  to  correct  the 
problems.  In  addition,  cooling  and  power  issues  with  the  radar  have 
contributed  to  overruns  with  the  prime  power  unit.  Numerous  fan  motor 
control  system  redesigns  and  retrofits  for  the  cooling  system  drove  costs 
by  the  supplier.  Inexperience  with  building  a  prime  power  unit  and  a 
limited  understanding  of  the  true  complexity  and  risks  associated  with  the 
system  led  to  significant  cost  growth  and  delivery  delays.  If  the  contractor 
continues  to  perform  as  it  did  through  September  2009,  our  analysis 
projects  that  at  completion  in  January  2011,  the  work  under  the  contract 
could  cost  from  $267.2  million  to  $287.4  million  more  than  the  budgeted 
cost  of  $4.8  billion. 


THAAD  Fire  Unit  Fielding  The  THAAD  fire  unit  fielding  production  contractor  overran  fiscal  year 
Production  2009  budgeted  cost  and  schedule  by  $4.7  million  and  $10.7  million, 

respectively.  The  fiscal  year  cost  and  schedule  overruns  were  caused 
primarily  by  the  missile  and  fire  control  components.  Unfavorable  missile 
cost  and  schedule  variances  were  the  result  of  hardware  failures 
associated  with  components  of  the  inertial  measurement  unit, 
communications  transponder,  and  the  boost  motor  causing  delays  and 
rework.  In  addition,  the  fire  control  portion  of  the  contract  experienced 
overruns  because  of  unplanned  engineering  design  changes  and  labor 
associated  with  fire  control  hardware  and  issues  identified  during  testing. 
These  changes  were  made  to  the  hardware  and  deliveries  already 
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completed.  See  figure  13  for  an  illustration  of  cumulative  cost  and 
schedule  variances  during  the  course  of  the  fiscal  year. 


Figure  13:  THAAD  Fire  Unit  Fielding  Production  Cumulative  Cost  and  Schedule  Performance 
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Despite  fiscal  year  overruns,  the  fire  unit  production  contractor  continues 
to  underrun  its  total  contract  cost  and  schedule.  The  contractor,  since  it 
began  performance  in  December  2006,  is  currently  $6.1  million  under 
budgeted  costs  and  has  completed  $11.3  million  more  worth  of  work  than 
originally  anticipated.  The  cost  underruns  are  primarily  due  to  a  slow 
start-up  on  fire  unit  fielding  level  of  effort  activities.  Schedule  variances 
are  not  reported  on  level  of  effort  activities,  so  delaying  these  activities 
would  save  on  costs  without  affecting  reported  schedule.  However,  these 
false  positive  cost  variances  will  erode  over  time  once  the  work  gets 
accomplished.  When  planned  level  of  effort  work  is  not  performed,  EVM 
metrics  are  distorted  because  they  show  cost  savings  for  work  that  has  not 
yet  been  accomplished.  However,  once  the  work  is  finished,  large 
unfavorable  cost  variances  will  be  revealed  since  the  program  will  need  to 
expend  funds  to  accomplish  the  work  for  which  it  has  already  received 
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credit.  In  addition,  the  program  reports  its  favorable  schedule  variances 
are  due  to  the  transfer  of  excess  interceptor  hardware  from  the 
development  contract  to  the  fire  unit  fielding  contract.  Although  the 
favorable  schedule  variance  from  this  transfer  of  hardware  is  nearly  $23.0 
million,  offsets  occurred  from  delayed  interceptor  build  activity  driven  by 
multiple  supplier  hardware  issues  and  schedule  delays  because  of  issues 
with  the  boost  motor  including  unplanned  replacement  of  motor  cases, 
delayed  case  fabrication,  and  slowed  operations  caused  by  a  safety 
incident  at  a  production  facility.  If  the  contractor  continues  to  perform  as 
it  did  through  September  2009,  our  analysis  projects  that  at  completion  in 
August  2011,  the  work  under  the  contract  could  cost  from  $1.3  million  to 
$17.9  million  less  than  the  budgeted  cost  of  $604.4  million.  However,  it 
should  be  noted  that  the  projection  of  the  estimated  cost  at  completion 
may  also  be  overestimated  because  it  is  based  on  current  cost 
performance  that  is  inflated  because  of  level  of  effort  activities  and 
schedule  performance  which  are  inflated  by  transfers  of  materials  from 
another  contract. 
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To  examine  the  progress  Missile  Defense  Agency  (MDA)  prime 
contractors  made  in  fiscal  year  2009  in  cost  and  schedule  performance,  we 
examined  contractor  performance  on  14  Ballistic  Missile  Defense  System 
(BMDS)  element  contracts.  In  assessing  each  contract,  we  examined 
contract  performance  reports  from  September  2008  through  October  2009 
for  each  contract,  including  the  format  1  variance  data  report,  cost  and 
schedule  variance  explanations  included  in  the  format  5,  and  format  2 
organizational  category  variance  totals  where  available.  We  performed 
extensive  analysis  on  the  format  1  of  the  contract  performance  reports  in 
order  to  aggregate  the  data  and  verify  data  reliability. 

To  ensure  data  reliability,  we  performed  a  series  of  checks  based  on 
consultation  with  earned  value  experts  and  in  accordance  with  GAO 
internal  reliability  standards.  We  began  by  tracking  which  earned  value 
management  (EVM)  systems  that  produced  the  contract  performance 
reports  were  compliant  with  American  National  Standards  Institute 
standards  in  2009  by  reviewing  the  certification  documentation.  We 
received  this  documentation  through  the  Defense  Contract  Management 
Agency  (DCMA),  which  performs  independent  EVM  surveillance  of  MDA 
contractors.  We  then  reviewed  the  latest  integrated  baseline  review  out- 
briefs  for  the  BMDS  elements’  contracts  to  examine  the  earned  value- 
related  risks  that  were  identified  during  the  review  and  followed  up  with 
the  program  office  to  see  which,  if  any,  risks  were  still  open  action  items. 

To  further  review  the  contract  performance  report  format  1  data,  we 
performed  basic  checks  on  the  totals  from  contract  performance  report 
format  1  to  ensure  that  they  matched  up  with  organizational  totals  from 
the  contract  performance  report  format  2,  where  available.  This  check 
enabled  us  to  review  whether  the  earned  value  data  were  consistent 
across  the  report.  In  addition,  we  obtained  a  spreadsheet  tool  from  GAO 
internal  earned  value  experts  to  perform  a  more  extensive  check  of  the 
data.  Using  this  tool,  we  ran  various  analyses  on  the  data  we  received  to 
search  for  anomalies.  We  then  followed  up  on  these  anomalies  with  the 
program  offices  that  manage  each  of  the  14  BMDS  element  contracts.  We 
reviewed  the  responses  with  GAO  EVM  experts  and  further  corroborated 
the  responses  with  DCMA  officials. 

We  used  contract  performance  report  data  in  order  to  generate  our 
estimated  overrun  or  underrun  of  the  contract  cost  completion  by  using 
formulas  accepted  by  the  EVM  community  and  printed  in  the  GAO  Cost 
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Estimating  and  Assessment  Guide.'  We  generated  multiple  formulas  for 
the  projected  contract  cost  at  completion  that  were  based  on  how  much  of 
the  contract  had  been  completed  up  to  September  2009.  The  ranges  in  the 
estimates  at  completion  are  driven  by  using  different  efficiency  indices 
based  on  the  program’s  completion  to  adjust  the  remaining  work 
according  to  the  program’s  past  cost  and  schedule  performance.  The  idea 
in  using  the  efficiency  index  is  that  how  a  program  has  performed  in  the 
past  will  indicate  how  it  will  perform  in  the  future.  In  close  consultation 
with  earned  value  experts,  we  reviewed  the  data  included  in  the  analysis 
and  made  adjustments  for  anomalous  data  where  appropriate. 

We  conducted  this  performance  audit  from  February  2010  to  July  2010  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives. 


1  GAO,  GAO  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and 
Managing  Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  Web  site  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  Web  site, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 

MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 

Public  Affairs 


Chuck  Young,  Managing  Director,  youngcl@gao.gov,  (202)  512-4800 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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